Local replanning of robot movements

ABSTRACT

Systems, methods, devices, and other techniques for planning and re-planning robot motions. The techniques can include obtaining a schedule that defines an execution order for a plurality of plans, each plan defining a sequence of tasks; providing instructions to the robotic system to execute plans from the plurality of plans according to the schedule; obtaining measurements for one or more parameters that represent a state of the robotic system or its environment that results from execution of at least one plan from the plurality of plans; determining, based on the measurements, that a particular plan from the plurality of plans, which has not completed execution, is to be revised; generating, using the measurements, a revised version of the particular plan; and adjusting the schedule so as to resolve a conflict between the revised version of the particular plan and at least one other plan in the plurality of plans.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of, and claims priorityto pending U.S. application Ser. No. 16/256,965, filed on Jan. 24, 2019.The entirety of the disclosure of the prior application is hereinincorporated by reference.

BACKGROUND

Robots have been employed in various settings and industries to automatemanufacturing and fabrication processes. Some systems have beendeveloped that provide multiple, individually functioning robotsconfigured to operate concurrently in a shared workspace. Multiplerobots in a robotic system can be programmed to execute tasks to achievea goal, such as the production of an item of manufacture, the assemblyof a device, the fabrication of a component, or the inspection of anobject. For example, in an automotive assembly line, robots witharticulating arms may surround a chassis of a vehicle and operate in theworkspace occupied by the chassis to attach components, apply machiningoperations (e.g., drilling, spot welding, milling, or grinding one ormore portions of the chassis), perform measurements and inspections, orperform a combination of these and other operations. Each robot, forexample, may pivot about a base, and may extend from the base with one,two, three, or more articulating segments, thereby allowing it to moveand reach a portion or entirety of the shared workspace. The base itselfmay be fixed, or may move in one or more dimensions, as well. Tocomplete a production goal, robots in a robotic system may performindividual tasks serially and/or concurrently with other robots, and maybe required to share not only a common workspace, but also resourcessuch as tools or supplies. A controller can coordinate the movements andexecution of tasks by robots that share such resources.

SUMMARY

This specification generally describes systems, methods, devices, andother techniques for planning the movements of robots, and morespecifically, to techniques for re-planning the movements of the robotsbased on real-time feedback about the robots' execution of a process.

In some implementations, a robotic system receives, from a schedulersystem, a schedule that defines an execution order for multiple plans,where each plan includes at least one task to be performed by therobotic system. At least one particular plan of the schedule includes aspeculation point. In general, a speculation point of a plan can referto a point in which the robotic system requires or can accept one ormore parameters in order to continue executing the plan. The parameterscan relate to or describe measurements that have been taken duringexecution of the schedule up to the speculation point (e.g.,measurements taken during or following previously executed plans and/ormeasurements taken during the current plan based on previously executedtasks up to the speculation point). The measurements can relate toresults of previously executed tasks or plans, such as dimensions orlocations of features in a workplace, for example. One or more robots ofthe robotic system executes the plans of the schedule until thescheduler system determines that the speculation point is reached, atwhich time further execution of the particular plan is contingent onadditional input to the robotic system. The additional input may requirethe scheduler system to revise the particular plan. If required, thescheduler system revises the particular plan to generate a revisedversion of the particular plan. If the particular plan is revised, thescheduler system checks the schedule to determine if the particular planconflicts with any other plans of the schedule. If there is no conflictdetected, the robotic system resumes execution of the schedule. If thescheduler system detects a conflict, then it adds a dependency betweenthe particular plan and the conflicting plan. The dependency requiresthe robotic system to complete execution of the particular plan beforebeginning execution of the conflicting plan.

In general, a first aspect includes a computer-implemented method thatincludes: obtaining a schedule that defines an execution order for aplurality of plans, each plan defining a sequence of tasks to beexecuted by a robotic system; providing instructions to the roboticsystem to execute plans from the plurality of plans according to theschedule; obtaining measurements for one or more parameters thatrepresent a state of the robotic system or its environment that resultsfrom execution of at least one plan from the plurality of plans;determining, based on the measurements for the one or more parametersthat represent the state of the robotic system or its environment, thata particular plan from the plurality of plans, which has not completedexecution, is to be revised; generating, using the measurements for theone or more parameters that represent the state of the robotic system orits environment, a revised version of the particular plan; and adjustingthe schedule that defines the execution order for the plurality of plansso as to resolve a conflict between the revised version of theparticular plan and at least one other plan in the plurality of plans.

In some embodiments, adjusting the schedule that defines the executionorder for the plurality of plans includes delaying execution of the atleast one other plan that conflicts with the revised version of theparticular plan by adding a dependency between the at least one otherplan and the revised version of the particular plan. The dependencyconditions execution of the at least one other plan on completion ofexecution of the revised version of the particular plan. In otherembodiments, adjusting the schedule that defines the execution order forthe plurality of plans comprises enforcing a constraint that fixes thesequences of tasks defined by each plan in the plurality of plans otherthan the particular plan.

In some embodiments, the computer-implemented method further includesproviding instructions to the robotic system to execute plans from theplurality of plans, including the revised version of the particularplan, according to the adjusted schedule.

In some embodiments, the computer-implemented method further includesadjusting the schedule that defines the execution order for theplurality of plans without revising any plans in the plurality of plansother than the particular plan.

In some embodiments, the computer-implemented method includesdetermining whether any conflicts in the schedule result from replacingthe particular plan with the revised version of the particular plan and,in response to determining that a conflict results between the revisedversion of the particular plan and at least one other plan in theplurality of plans, adding a dependency between the revised version ofthe particular plan and the at least one other plan so as to resolve theconflict.

In some embodiments, the computer-implemented method of claim 3, whereindetermining whether any conflicts in the schedule result from replacingthe particular plan with the revised version of the particular plancomprises determining whether the schedule would require concurrentexecution of two or more plans that make use of a common resource.

In some embodiments, the common resource is space, fuel, or a tool.

In some embodiments, an initial version of the particular plan and therevised version of the particular plan have at least one of differentexecution times or different resource requirements.

In some embodiments, the obtained schedule is an initial version of theschedule and the method further includes generating the initial versionof the schedule based on at least one of expected execution times forthe plurality of plans or expected resource requirements for theplurality of plans. At least one of an expected execution time or anexpected resource requirement for the particular plan is based onexpected values for the one or more parameters that represent thecurrent state of the robotic system or its environment.

In some embodiments, the measurements for the one or more parametersthat represent the current state of the robotic system or itsenvironment are different from the expected values for the one or moreparameters.

In some embodiments, the one or more parameters that represent thecurrent state of the robotic system or its environment include a firstparameter that represents a result of executing one or more tasks in theparticular plan before the particular plan completes execution.

In some embodiments, the result of the one or more tasks includes alocation or a dimension of a component that was made or placed on aproduct by a robotic system in the course of execution of one or moreplans.

In some embodiments, the robotic system includes a plurality of robotsconfigured to work cooperatively in an automated process to fabricate aproduct.

Other implementations of this and other aspects include correspondingsystems and one or more non-transitory computer-readable media havinginstructions stored thereon that, when executed by one or moreprocessors, cause performance of the actions of the computer implementedmethods.

Some implementations of the subject matter disclosed herein can, incertain instances, achieve one or more of the following advantages. Aschedule defining an execution order for multiple plans can bepre-generated, without having to anticipate each potential conflict thatmay arise during the execution of the multiple plans. The process ofaccounting for each potential conflict may be computationally infeasibleif there are a large number of plans. Therefore, pre-generating aschedule represents a feasible alternative with greatly reducedcomputational complexity. Furthermore, when input to the robotic systemis received, the disclosed subject matter allows for one particular planto be revised locally, without requiring a complex rescheduling of eachplan in the schedule.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schedule of plans that can be executed by a robotic systemincluding plans A, B, C, and D.

FIG. 1B is a schedule that shows a revised version of plan C and aconflict arising from plan C being revised.

FIG. 1C is a schedule that shows a dependency that is added betweenplans B and the revised version of plan C.

FIG. 1D is a schedule free of the conflict shown in FIG. 1B.

FIG. 2 is a block diagram showing an example scheduler system and arobotic system.

FIG. 3 is an example flow diagram for adjusting a schedule to resolveconflicts between plans.

DETAILED DESCRIPTION

This specification generally describes systems, methods, devices, andother techniques for planning the movements of robots, and morespecifically, to techniques for re-planning the movements of the robotsbased on real-time feedback about the robots' execution of a process.

One or more robots can receive a schedule that describes an orderedsequence of movements to be performed by the robots. In general, eachschedule can describe one or more plans, and each plan describes one ormore tasks for the robots to perform. For example, a schedule caninclude a set of plans related to assembling components of a car. Afirst plan can relate to attaching the driver's side door of the car,while a second plan can relate to attaching the passenger's side door.Following this example, the tasks of the first and second plans caninclude a series of bolting actions that together attach the doors totheir respective sides. While the first and second plans may include thesame or similar tasks, they may be included in different plans. Forexample, the tasks may be separated into different plans based on therelative location of tasks, e.g., a first plan may include one or moretasks that are to be performed in a first location, while a second planmay include one or more tasks that are to be performed in a secondlocation. As another example, the tasks may be separated because giventwo of the same tool, the two plans can be performed concurrently by twodifferent robots.

In some cases, a plan can include a speculation point, which representsa task or a point in time in which further execution of the plan iscontingent on additional input to the robotic system. Based on theadditional input received, the scheduler system may revise theparticular plan and then resolve any conflicts that may arise betweenthe revised version of the particular plan and other plans of theschedule, which have not yet finished/completed execution.

Referring now to FIGS. 1A-1D, a series of schedules are depicted for anexample scenario that involves generating a revised version of a plan,detecting conflicts, and delaying a plan to resolve a detected conflict.

Each of FIGS. 1A-1D illustrates a different schedule, i.e., schedules100A through 100D, respectively. Boxes A, B, C, D of FIGS. 1A-1D eachrepresent a plan. The positions of the plans along the horizontal axisshows the relative timing of each plan. Although not explicitly shownwith respect to each plan of the schedules 100A-100D, each plan includesa series of tasks to be performed by the robotic system.

Dependencies between plans are represented by arrows between boxes. Twoplans are said to be dependent, if the robotic system must execute oneof the two plans before the other. That is, if a second plan isdependent on a first plan, then the robotic system must execute thefirst plan before executing the second plan. The dependencies betweenplans can be caused by, for example, a resource requirement that isshared by two plans, e.g., if the execution of a second plan requires aresource, which the execution of the first plan also requires, then therobotic system cannot execute the second plan until it executes thefirst plan.

A resource used by the robotic system can include physical resources,such as supplies or a tool. A resource can also include physical space,such as space in the environment of the robot, for example, in awarehouse or production facility.

If instead two plans are independent of one another, then the roboticsystem can execute the two plans in any order or at the same time. Forexample, two plans may be independent of one another if they can each beexecuted independently of one another (e.g., with different robots ofthe robotic system and without conflicts that would arise fromconcurrent use of a shared resource).

FIG. 1A is a schedule 100A of plans that can be executed by a roboticsystem. Schedule 100A includes plans A, B, C, and D. Plan C includes twoparts, C_(head) and C_(tail). Plan B is dependent on plan A, while planD is dependent on plan C, as shown by the arrows that connect the plans.Because plan B is dependent on plan A, the robotic system executes planB only after executing plan A. Similarly, plan D is dependent on plan C,with plan D executing only after plan C is executed.

Schedule 100A also shows that plans A and B are independent of plans Cand D. In some cases, independent plans can be executed at the sametime. For example, plan A can be executed at the same time as plan C, asshown by an overlap between the two plans with respect to their verticalpositions. Similarly, schedule 100A shows that plans B and D areexecuted concurrently.

In general, the starting and ending times of plans are not set inadvance by a schedule. Rather, the starting and ending times are basedon the dependencies between plans, i.e., the time at which a particularplan becomes executable by a robot based on, for example, what plans areto be executed before the execution of the particular plan.

Plan C includes a speculation point 102, shown by a dashed line. Ingeneral, a speculation point of a plan identifies a point in which therobotic system requires or can accept one or more parameters in order tocontinue executing the plan. In some implementations, the one or moreparameters represent a current state of the robotic system or itsenvironment. For example, the one or more parameters can represent anactual dimension of a part or product being fabricated, which may bemore or less than a nominal dimension specified in a schematic for thepart or product. In another example, the parameters can represent aquantity of a particular resource that is required for the execution ofthe tasks of a plan, e.g., a number of bolts available to a robot beforethe robot begins executing a series of bolting tasks.

While the robotic system executes plan C, the robotic system uses one ormore sensors to determine measurements for one or more parameters thatrepresent a current state of the robotic system or its environment. Therobotic system communicates the measurements to a scheduler system,which determines the current state of the robotic system or itsenvironment. Speculation point 102 represents a time at which thescheduler system determines, based on the current state of the roboticsystem or its environment, that plan C may need to be re-planned.

In response to determining that plan C should be re-planned, thescheduler system generates a revised version of plan C. FIG. 1B is aschedule 100B that shows plans A, B, and D. Schedule 100B also showsplan C′, which is the revised version of plan C. Like plan C, plan C′includes C_(head), because the robotic system executes the tasksincluded in C_(head) prior to speculation point 102. Instead ofincluding C_(tail), C′ includes C_(tail)′, which is the revised versionof G_(tail). C_(tail)′ includes additional tasks, resulting in the plantaking longer to be executed than C_(tail), as shown by the plan's widerprofile with respect to the horizontal axis. When the scheduler systemgenerates C′, the dependencies that existed between C and the otherplans of schedule 100A, i.e., plan D, are inherited by plan C′.

In some implementations, an initial version of a plan and a revisedversion of the plan differ in resource requirements. For example, if afirst plan and a second plan both require the use of a first paintsprayer, then a revised version of the second plan can differ from theinitial version of the second plan by requiring the use of a secondpaint sprayer.

FIG. 1B also shows a conflict arising from plan C being revised. Unlikeplans A and C, which can be executed concurrently, plan B cannot beexecuted at the same time as plans C or C′. The portion of plan B whichoverlaps with plan C′, with respect to the horizontal positions of thetwo plans, is shown as conflict 104.

To ensure that the robotic system can execute both the tasks of plan Band the tasks of plan C′, the scheduler system resolves conflict 104 byadding a dependency between the conflicting plans. Referring to FIG. 1C,a schedule 100C includes a dependency 106, which the scheduler systemadds between plans B and C′. More specifically, dependency 106illustrates that plan B depends on plan C′.

With the addition of dependency 106, plan B is dependent on plan A andplan C′. Therefore, the robotic system must execute both plan A and planC′ before executing plan B. Because of the addition of dependency 106,the execution of plan B is delayed until after plan C′. Referring toFIG. 1D, a schedule 100D after conflict 104 has been resolved by theaddition of dependency 106.

FIG. 1D also shows that no other plan, apart from plan C, is revised inlight of the scheduler system determining that plan C includesspeculation point 102. That is, the scheduler system can be configuredto resolve conflicts without revising plans that do not include aspeculation point. Since the scheduler system can be configured toresolve conflicts arising during execution of a process, the schedulercan avoid the expense of re-generating optimized plans from scratch toresolve a conflict, and can instead perform more modest schedulingadjustments that involve re-scheduling existing plans without revisingthe tasks or sequence of tasks within each plan. For example, comparingschedules 100A and 100D, plans A, B, and D are not affected by the localre-planning that that scheduler system performs in response to conflict104.

FIG. 2 shows a block diagram of an example scheduler system 200 incommunication with a robotic system 260. For example, robotic system 260can include one or more robots configured to work cooperatively in anautomated process to fabricate a product. The scheduler system 200includes an execution manager 210, which is communicatively coupled to ascheduler subsystem 220, a sensing and measurement subsystem 230, are-scheduler subsystem 240, and a conflict detector 250. In addition toshowing the components of the scheduler system 200 and the roboticsystem 260, FIG. 2 also shows stages A through G, which illustrate aflow of data.

Scheduler subsystem 220 is configured to receive data representing a setof tasks to be performed by robotic system 260, and to output a scheduleto be executed by robotic system 260. As discussed above, each scheduleincludes one or more plans, while each plan includes one or more tasks.Each task corresponds to an action to be completed by robotic system260. For example, a task of a plan can include performing a physicalaction, e.g., a first task can be riveting components, while a secondtask can be spray painting the riveted components. Accordingly, roboticsystem 260 may require a certain tool in order to complete a particulartask. Following the previous example, robotic system 260 may require arivet tool to complete the first task, while the system may require apaint sprayer and paint to complete the second task.

Scheduler system 200 also includes sensing and measurement subsystem140, which is configured to interface with one or more sensors of therobotic system 260. Sensing and measurement subsystem 230 receivesmeasurements for one or more parameters and determines the current stateof robotic system 260 based on the measurements. Sensing and measurementsubsystem 230 can communicate data representing the current state ofrobotic system 260 to re-scheduler subsystem 240.

Re-scheduler subsystem 240 is configured to receive data representingthe current state of robotic system 260 and to generate a revisedversion of a plan based on the current state. Re-scheduler subsystem 240is also configured to generate a revised schedule that includes one ormore delayed plans.

The revised version of the plan generated by re-scheduler subsystem 240may conflict with one or more other plans in the schedule. Conflictdetector 250 is configured to identify conflicts between the revisedplan and one or more other plans in the schedule. If conflict detector250 identifies a conflict, re-scheduler subsystem 240 adjusts theschedule to resolve the conflict. For example, if the execution of afirst plan would conflict with the execution of a second plan, thenre-scheduler subsystem 240 can generate a revised schedule in which thesecond plan is delayed, so that the first and second plans no longerconflict.

In the example of FIG. 2, during stage A, scheduler system 220 generatesa schedule, which includes plans to be executed by robotic system 260,and communicates the schedule to execution manager 210.

In stage B, execution manager 210 communicates the schedule to roboticsystem 260. After receiving the schedule, robotic system 260 executesthe schedule. That is, robotic system 260 performs the tasks of theplans included in the schedule. Robotic system 260 performs the tasksuntil it arrives at a speculation point, at which point the roboticsystem can accept additional information in order to proceed with theexecution of the schedule. For example, the robotic system may executethe tasks of a current plan while communicating to scheduler system 220measurements for one or more parameters that represent a current stateof the robotic system or its environment. In response to themeasurements received at a speculation point, scheduler subsystem 220determines that the execution of the current plan requires additionalresources.

In stages C/D, robotic system 260 uses one or more sensors to determinemeasurements for one or more parameters that represent the current stateof the robotic system or its environment. Robotic system 260communicates the measurements to scheduler system 200, e.g., bycommunicating the measurements to execution manager 210. Sensing andmeasurement subsystem 230 receives the measurements, e.g., fromexecution manager 210, and determines the current state of roboticsystem 260 or its environment. For example, the measurements canindicate a quantity of the second resource available.

In stage E, sensing and measurement subsystem 230 communicates datarepresenting the current state of robotic system 260 or its environmentto re-scheduler subsystem 240. Using this information, re-schedulersubsystem 240 determines if the particular plan should be revised, andif so, the re-scheduler subsystem generates a revised version of theparticular plan based on the measurements for the one or moreparameters. The tasks of the revised version of the particular planreplace the tasks of the particular plan at a speculation point of theparticular plan.

As an example, the measurements can indicate the current quantity of thesecond resource available. Re-scheduler subsystem 240 can determine thatthe current quantity of the second resource is insufficient to performthe tasks that follow the speculation point. Therefore, re-schedulersubsystem 240 can revise the particular plan to include tasks such asmoving robotic system 260 to a resource repository, retrieving aquantity of the second resource sufficient to perform one or more tasks,and returning the robotic system to a workstation to perform the one ormore tasks.

As another example, the particular plan may assume that a part requiredfor the execution of the particular plan, is at a first location. Themeasurements can indicate that the part is not at the first location,but rather, at a second location. In response to determining that thepart is at the second location, re-scheduler subsystem 240 can generatea revised version of the particular plan having a modified toolpath forthe robotic system, to account for the location of the part.

If, in stage E, re-scheduler subsystem 240 determines that theparticular plan should be revised, then it generates a revised versionof the particular plan. In stage F, re-scheduler subsystem 240communicates the revised plan to conflict detector 250, which determineswhether the revised plan conflicts with any other plans in the schedule.

In stage G, conflict detector 250 communicates any conflicts detected tore-scheduler subsystem 240. Prior to updating the schedule to includethe revised plan, re-scheduler subsystem 240 adjusts the schedule basedon the conflicts detected by re-scheduler subsystem 240. For example, ifconflict detector 250 determines that there is a conflict between therevised version of the particular plan and a conflicting plan, thenre-scheduler subsystem 240 can resolve the conflict by adding adependency between the revised version of the particular and theconflicting plan so that the robotic system executes the conflictingplan only after executing the revised version of the particular plan.Therefore, by modifying the dependency structure of the schedule,re-scheduler system 240 can resolve a conflict without re-planning plansother than the particular plan.

Once all conflicts are detected and resolved by conflict detector 250and re-scheduler subsystem 240, the reschedule subsystem communicatesthe conflict-free schedule to execution manager 210. In turn, executionmanager 210 communicates the schedule to robotic system 260, whichexecutes each plan of the schedule.

FIG. 3 is an example flow diagram of an example process 300 foradjusting a schedule of plans to resolve a conflict between plans. Theexample process can be performed by scheduler system 200 of FIG. 2.

The scheduler system obtains a schedule that defines an execution orderfor a plurality of plans, each plan defining a sequence of tasks to beexecuted by a robotic system (305). For example, the scheduler systemcan include a scheduler subsystem that is configured to generate aschedule based on the tasks to be performed by a robotic system.

In some implementations, the scheduler subsystem can generate theschedule based on at least one of expected execution times for theplurality of plans or expected resource requirements for the pluralityof plans. For example, the scheduler subsystem can determine theexpected execution times and/or expected resource requirements fromhistorical data, e.g., execution of previous plans.

The scheduler system provides instructions to the robotic system toexecute plans from the plurality of plans according to the schedule(310). The scheduler system can communicate the schedule to the roboticsystem, which can execute the plans according to the schedule. That is,the robotic system can execute the tasks that make up the plans of theschedule.

The scheduler system obtains measurements for one or more parametersthat represent a current state of the robotic system or its environment(315). The robotic system uses one or more sensors to obtain themeasurements, which it communicates to the scheduler system. Using themeasurements, the scheduler system determines the current state of therobotic system or the current state of the environment of the roboticsystem.

The scheduler system determines that a particular plan requiresadjustments based on the measurements for the one or more parameters(320).

In some implementations, the one or more parameters that represent thecurrent state of the robotic system or its environment includes a firstparameter that represents a result of one or more tasks that wereexecuted by the robotic system prior to or during the particular plan.As an example, the action may be using a paint sprayer and the firstparameter can represent the amount of paint left in the paint sprayer, ameasurement of how much paint was applied, and an actual area of anobject where the paint was applied. The scheduler system can determinethat the particular plan requires adjustments if the amount of paint isinsufficient to complete a task or if the area where the paint wasapplied was incorrect.

In some implementations, the result of the one or more tasks includes alocation or a dimension of a component that was made or placed on aproduct by a robotic system in the course of execution of one or moreplans. For example, if the robotic system was moving an object prior tothe particular plan, the first parameter can represent the location ofthe object before the particular plan. As another example, if therobotic system was cutting a sheet of metal, the first parameter canrepresent the dimensions of the sheet of metal after the robotic systemis finished cutting it. If the measured location or dimension differsfrom an expected value that the scheduler system used to generate theparticular plan, the scheduler system can determine that theparticularly plan requires adjustments in light of the measuredparameters.

The scheduler system generates a revised version of the particular planbased on the measurements for the one or more parameters (325). Thescheduler system can include a scheduler subsystem that is configured toreceive the measurements for the one or more parameters and determinethe current state of the robotic system or the environment of therobotic system. Using the current state of the robotic system or itsenvironment, the scheduler subsystem generates a revised version of theparticular plan.

For example, the current state of the robotic system can include thepositions of one or more robots of the robotic system or the positionsof one or more components of the one or more robots, e.g., the positionof an arm of a robot.

The current state of the robotic system can further include a quantityof a material or resource required to perform one or more tasks of theschedule. The current state of the environment of the robot can includethe presence of one or more tools required by the robotic system. Thecurrent state of the environment can also include the presence of one ormore parts to be worked on by the robotic system, e.g., if the tasks ofthe schedule include assembling an electronic device, then the currentstate of the environment can indicate the presence of the electronicdevice.

The current state of the environment of the robot can further includeinformation related to the space occupied by other robots executingplans around the same time as a particular plan that is to bere-planned. The current state of the environment during re-planning maydiffer from the state of the environment that was initially used by thescheduler system to generate an initial schedule.

In some implementations, the scheduler system generates at least one ofan expected execution time or an expected resource requirement for theparticular plan based on expected values for the one or more parametersthat represent the current state of the robotic system or itsenvironment. The measurements for the one or more parameters, asobtained in stage 320, may be different from the expected values for theone or more parameters.

The scheduler system determines whether any conflicts in the scheduleresult from replacing the particular plan with the revised version ofthe particular plan (330). Replacing the particular plan with therevised version of the particular plan may cause a conflict between therevised version of the particular plan and a plan that is independent ofthe revised version of the particular plan. For example, the schedulemay require concurrent execution of the particular plan and the revisedversion of the particular plan.

As another example, the revised version of the particular plan mayinclude a different starting position or ending position of the roboticsystem. Therefore, the revised version of the particular plan mayconflict with one or more other plans of the schedule.

As another example, a conflict may arise as a result of the schedulerequiring concurrent execution of two or more plans that make use of acommon resource. In some implementations, the common resource can bespace, fuel, or a tool. For example, a conflict would arise if two ormore plans both require a paint sprayer, while both are scheduled toexecute at the same time.

The scheduler system adjusts the schedule that defines the executionorder for the plurality of plans so as to resolve a conflict between therevised version of the particular plan and at least one other plan inthe plurality of plans (335). For example, a re-scheduler subsystem ofthe scheduler system can resolve a conflict between the revised versionof the particular plan and another, conflicting plan by delaying thestart of the conflicting plan, as described in greater detail withregard to FIG. 4. As another example, adjusting the schedule can includedelaying execution of the conflicting plan by adding a constraint to theexecution order of the schedule such that the particular plan must becompleted before the execution of the conflicting plan.

While, in stage 325, the scheduler system revises the particular plan,the scheduler system adjusts the schedule without revising any plansother than the particular plan.

Embodiments of the invention and all of the functional operationsdescribed in this specification may be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe invention may be implemented as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer-readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium may be a non-transitory computer readable storage medium, amachine-readable storage device, a machine-readable storage substrate, amemory device, a composition of matter effecting a machine-readablepropagated signal, or a combination of one or more of them. The term“data processing apparatus” encompasses all apparatus, devices, andmachines for processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus may include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

An “engine” as used herein refers to a data processing apparatus (e.g.,one or more computers, one or more processors) configured to performoperations. The data processing apparatus can be provided in one or morelocations, and can assume a centralized or distributed architecture.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any form of programminglanguage, including compiled or interpreted languages, and it may bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program may be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programmay be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer may be embedded inanother device, e.g., a tablet computer, a mobile telephone, a personaldigital assistant (PDA), a mobile audio player, a Global PositioningSystem (GPS) receiver, to name just a few. Computer readable mediasuitable for storing computer program instructions and data include allforms of non-volatile memory, media, and memory devices, including byway of example semiconductor memory devices, e.g., EPROM, EEPROM, andflash memory devices; magnetic disks, e.g., internal hard disks orremovable disks; magneto optical disks; and CD ROM and DVD-ROM disks.The processor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments of the invention maybe implemented on a computer having a display device, e.g., a CRT(cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user may provide input to thecomputer. Other kinds of devices may be used to provide for interactionwith a user as well; for example, feedback provided to the user may beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user may be received in anyform, including acoustic, speech, or tactile input.

Embodiments of the invention may be implemented in a computing systemthat includes a back end component, e.g., as a data server, or thatincludes a middleware component, e.g., an application server, or thatincludes a front end component, e.g., a client computer having agraphical user interface or a Web browser through which a user mayinteract with an implementation of the invention, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system may be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments may also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment mayalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination may in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems maygenerally be integrated together in a single software product orpackaged into multiple software products.

In each instance where an HTML file is mentioned, other file types orformats may be substituted. For instance, an HTML file may be replacedby an XML, JSON, plain text, or other types of files. Moreover, where atable or hash table is mentioned, other data structures (such asspreadsheets, relational databases, or structured files) may be used.

Thus, particular implementations of the invention have been described.Other implementations are within the scope of the following claims. Forexample, the actions recited in the claims may be performed in adifferent order and still achieve desirable results.

1-20. (canceled)
 21. A computer-implemented method, comprising:obtaining a schedule having a first plan and a second plan, each plandefining a sequence of tasks to be executed by a robotic system;obtaining sensor input representing a state of the robotic system duringexecution of the first plan; determining, based on the sensor input,that the first plan requires an adjustment; in response, generating anadjusted first plan; determining that the adjusted first plan conflictswith the second plan; and in response to determining that the adjustedfirst plan conflicts with a second plan, providing instructions to therobotic system to resume execution using the adjusted first plan and toexecute the second plan after the adjusted first plan has completed. 22.The method of claim 21, further comprising adding a dependency betweenthe adjusted first plan and the second plan in a schedule of plans. 23.The method of claim 21, wherein generating the adjusted first plancomprises adding one or more additional tasks to the first plan.
 24. Themethod of claim 23, wherein providing instructions to the robotic systemcomprises providing instructions to execute the second plan only afterthe one or more additional tasks have been executed.
 25. The method ofclaim 21, wherein the sensor input represents a quantity of an itemrequired to perform the first plan.
 26. The method of claim 25, whereinthe adjusted first plan includes a task to obtain additional itemsrequired to perform the first plan.
 27. The method of claim 21, whereindetermining that the adjusted first plan conflicts with the second plancomprises determining that the adjusted first plan and the second planwould compete for a common resource.
 28. The method of claim 27, whereinthe common resource is a workspace volume, a tool, or fuel.
 29. Themethod of claim 21, wherein obtaining sensor input comprises obtainingsensor input at a speculation point in the first plan, and whereinresuming execution using the adjusted first plan comprises resumingexecution from the speculation point.
 30. A system comprising: one ormore computers and one or more storage devices storing instructions thatare operable, when executed by the one or more computers, to cause theone or more computers to perform operations comprising: obtaining aschedule having a first plan and a second plan, each plan defining asequence of tasks to be executed by a robotic system; obtaining sensorinput representing a state of the robotic system during execution of thefirst plan; determining, based on the sensor input, that the first planrequires an adjustment; in response, generating an adjusted first plan;determining that the adjusted first plan conflicts with the second plan;and in response to determining that the adjusted first plan conflictswith a second plan, providing instructions to the robotic system toresume execution using the adjusted first plan and to execute the secondplan after the adjusted first plan has completed.
 31. The system ofclaim 30, wherein the operations further comprise adding a dependencybetween the adjusted first plan and the second plan in a schedule ofplans.
 32. The system of claim 30, wherein generating the adjusted firstplan comprises adding one or more additional tasks to the first plan.33. The system of claim 32, wherein providing instructions to therobotic system comprises providing instructions to execute the secondplan only after the one or more additional tasks have been executed. 34.The system of claim 30, wherein the sensor input represents a quantityof an item required to perform the first plan.
 35. The system of claim34, wherein the adjusted first plan includes a task to obtain additionalitems required to perform the first plan.
 36. The system of claim 30,wherein determining that the adjusted first plan conflicts with thesecond plan comprises determining that the adjusted first plan and thesecond plan would compete for a common resource.
 37. The system of claim36, wherein the common resource is a workspace volume, a tool, or fuel.38. The system of claim 30, wherein obtaining sensor input comprisesobtaining sensor input at a speculation point in the first plan, andwherein resuming execution using the adjusted first plan comprisesresuming execution from the speculation point.
 39. One or morenon-transitory computer storage media encoded with computer programinstructions that when executed by one or more computers cause the oneor more computers to perform operations comprising: obtaining a schedulehaving a first plan and a second plan, each plan defining a sequence oftasks to be executed by a robotic system; obtaining sensor inputrepresenting a state of the robotic system during execution of the firstplan; determining, based on the sensor input, that the first planrequires an adjustment; in response, generating an adjusted first plan;determining that the adjusted first plan conflicts with the second plan;and in response to determining that the adjusted first plan conflictswith a second plan, providing instructions to the robotic system toresume execution using the adjusted first plan and to execute the secondplan after the adjusted first plan has completed.
 40. The one or morecomputer storage media of claim 39, wherein the operations furthercomprise adding a dependency between the adjusted first plan and thesecond plan in a schedule of plans.